The field of data work has become increasingly complex, requiring a range of skills that are often inaccessible to non-technical experts. Automated Machine Learning (AutoML) technology has emerged as a solution to this problem by automating certain aspects of data work, such as model selection and data preparation. However, AutoML still requires human intervention to be functional, resulting in a complex and collaborative process that can be difficult to trace. To address this issue, researchers have constructed a taxonomy of data work artifacts that captures both the human and machine-generated processes involved in AutoML. The taxonomy was developed through an extensive literature review spanning Machine Learning, Human Computer Interaction, Computer Supported Collaborative Work, Information Visualization, and Visual Analytics. The research team gathered an initial set of 13 papers before identifying a systematic set of published research and pre-prints on AutoML. They then conducted eight iterations to develop the taxonomy by reading the literature sources, extracting artifacts that met the definition of the meta characteristic, classifying those items, and grouping them according to an evolving set of artifact properties. The resulting taxonomy is concise, robust, comprehensive, extensible and explanatory. It comprises multiple interrelated phases that leverage statistical and computational techniques for data preparation, analysis, deployment and communication. The taxonomy also includes artifact identification and classification methods for capturing artifacts with their properties and dependencies. To operationalize the taxonomy further, the researchers developed AutoMLTrace - a visual interactive sketch showing both the context and temporality of human-ML/AI collaboration in data work. They demonstrated its utility via a usage scenario with an enterprise software development team. Finally, reflecting on their approach to developing the taxonomy through a broad literature review but assessing its utility primarily through a collaboration with one team; they acknowledge more work is required to assess its generality but remain optimistic about its potential for developing data visualization tools that interrogate sociotechnical relationships in automated data work.
Data work is a complicated job that needs special skills. Automated Machine Learning (AutoML) technology helps with some parts of data work, but people still need to help for it to work well. Researchers made a list of all the things people and machines do in AutoML called a taxonomy. They did this by reading lots of books and papers about it. The taxonomy has different steps that use math and computers to prepare, analyze, deploy, and communicate data. The researchers also made a picture called AutoMLTrace to show how people and machines work together in data work.
Exploring Automated Machine Learning (AutoML) Through a Taxonomy of Data Work Artifacts
Data work has become increasingly complex, requiring a range of skills that are often inaccessible to non-technical experts. To address this issue, researchers have developed an automated machine learning (AutoML) technology that automates certain aspects of data work such as model selection and data preparation. However, AutoML still requires human intervention to be functional, resulting in a complex and collaborative process that can be difficult to trace. To better understand the sociotechnical relationships involved in AutoML, the research team constructed a taxonomy of data work artifacts that captures both the human and machine-generated processes involved in AutoML.
Developing the Taxonomy
The research team conducted an extensive literature review spanning Machine Learning, Human Computer Interaction, Computer Supported Collaborative Work, Information Visualization and Visual Analytics. From this review they gathered an initial set of 13 papers before identifying a systematic set of published research and pre-prints on AutoML. They then conducted eight iterations to develop the taxonomy by reading these sources for artifacts that met their definition of meta characteristics; classifying those items; and grouping them according to an evolving set of artifact properties.
The resulting taxonomy is concise yet robust with comprehensive coverage across multiple interrelated phases including statistical techniques for data preparation; analysis; deployment; communication; artifact identification/classification methods for capturing artifacts with their properties/dependencies; plus visual interactive sketches showing context/temporality for human-machine collaboration in data work.
Demonstrating Utility Through Usage Scenario
To operationalize the taxonomy further, the researchers developed AutoMLTrace - a visual interactive sketch showing both the context & temporality of human-machine collaboration in data work - which they demonstrated via usage scenario with enterprise software development team. The results were promising but more assessment is needed to assess its generality before it can be used as tool for developing visualization tools interrogating sociotechnical relationships in automated data work.
Conclusion
In conclusion, this paper presents a detailed taxonomy outlining both human & machine generated processes involved in Automated Machine Learning (AutoML). The authors demonstrate its utility through usage scenarios while acknowledging more assessment is needed before it can be used as tool for developing visualization tools interrogating sociotechnical relationships in automated data work
Automated Machine Learning (AutoML) technology can lower barriers in data work yet still requires human intervention to be functional. However, the complex and collaborative process resulting from humans and machines trading off work makes it difficult to trace what was done, by whom (or what), and when. In this research, we construct a taxonomy of data work artifacts that captures AutoML and human processes. We present a rigorous methodology for its creation and discuss its transferability to the visual design process. We operationalize the taxonomy through the development of AutoMLTrace, a visual interactive sketch showing both the context and temporality of human-ML/AI collaboration in data work. Finally, we demonstrate the utility of our approach via a usage scenario with an enterprise software development team. Collectively, our research process and findings explore challenges and fruitful avenues for developing data visualization tools that interrogate the sociotechnical relationships in automated data work.
