Essential for many industries ranging from Hollywood computer -generated images to product design, 3D modeling tools often use text or image prompts to dictate different aspects of visual appearance, such as color and form. Although this is logical as the first point of contact, these systems are still limited in their realism because of their negligence of something central to human experience: touch.
Tactile properties are fundamental for the uniqueness of physical objects, such as roughness, bump or the feeling of materials such as wood or stone. Existing modeling methods often require advanced computer -assisted design expertise and rarely support tactile comments that can be crucial for the way we perceive and interact with the physical world.
In this spirit, IT researchers in computer science and artificial intelligence laboratory (CSAIL) have created a new system to stylize 3D models using image prompts, effectively reproducing both visual appearance and tactile properties.
The CSAIL team's “tactstyle” tool allows creators to stylize 3D models based on images while incorporating the expected tactile properties of textures. TactStyle separates visual and geometric stylization, allowing the replication of visual and tactile properties from a single image input.
The “TactStyle” tool allows creators to stylize 3D models based on images while incorporating the tactile properties expected from textures.
Doctoral student Faraz Faruqi, the main author of a new article on the project, said that TactStyle could have large -scale applications, extending from interior decoration and personal accessories, including tactile learning tools. Tactstyle allows users to download a basic design – such as a Thingiverse helmet support – and personalize it with the styles and textures they want. In education, learners can explore various textures from around the world without leaving the classroom, while in product design, rapid prototyping becomes easier as designers quickly imprint several iterations to refine tactile qualities.
“You can imagine using this type of system for common objects, such as telephone supports and headphones, to allow more complex textures and improve tactile comments in various ways,” explains Faruqi, who co-written the document alongside the associate professor of MIT Stefanie Mueller, leader of the Human-Bordor Interaction Engineering Group (HCI). “You can create tactile educational tools to demonstrate a range of different concepts in fields such as biology, geometry and topography.”
The traditional methods of replication of textures imply the use of specialized tactile sensors – as a gelsight, developed at the MIT – which physically touch an object to capture its surface microgeometry as a “height field”. But this requires having a physical object or its surface recorded for replication. Tactstyle allows users to reproduce surface microtometry by taking advantage of generative AI to generate a field of height directly from an image of the texture.
In addition to that, for platforms like the 3D Thingiverse printing repository, it is difficult to take individual conceptions and personalize them. Indeed, if a user does not have a sufficient technical background, the design change manually covers the risk of “breaking” it so that it can no longer be printed. All these factors have encouraged Faruqi to question the construction of a tool which allows the personalization of downloadable models at a high level, but which also preserves functionality.
In experiences, Tactstyle has shown significant improvements in relation to traditional stylization methods by generating precise correlations between the visual image of a texture and its field of height. This allows the replication of touch properties directly from an image. A psychophysical experience has shown that users perceive the generated textures of tactStyle as similar to both tactile properties expected from the visual entry and the tactile characteristics of the original texture, leading to a unified tactile and visual experience.
Tactstyle uses a preexisting method, called “Style2fab»To modify the color channels of the model to match the visual style of the input image. Users first provide an image of the desired texture, then a refined variational autoencoder is used to translate the input image into a corresponding height field. This height field is then applied to modify the geometry of the model to create the touch properties.
The color and geometry stylization modules work in tandem, styling both the visual and tactile properties of the 3D model from a single image input. Faruqi says that the main innovation lies in the geometry stylization module, which uses a refined diffusion model to generate field fields from texture images – something that previous stylization frames do not reproduce with precision.
For the future, Faruqi says that the team aims to extend the tact style to generate new 3D models using a generative AI with integrated textures. This requires exactly exploring the type of pipeline necessary to reproduce both the shape and function of the 3D models manufactured. They also plan to study the “visu-haptic inadequacies” to create new experiences with materials that challenge conventional expectations, as something that seems to be made of marble but which has the impression of being made of wood.
Faruqi and Mueller co-owned the new newspaper alongside doctoral students Maxine Perroni-Scharf and Yunyi Zhu, a first cycle student in visit Jaskaran Singh Walia, a master's degree in visiting Shuyue Feng, and the assistant teacher Donald Degraen technology of the Human Interface Technology (HIT) Lab NZ in New Zealand.
