Research

This page describes recent research projects, and contains links to the relevant papers. You might also be interested in my page on Google Scholar or in my résumé.

Intelligent Annotation Dialogs

Intelligent Annotation Dialog

Learning Intelligent Dialogs for Bounding Box Annotation

We introduce Intelligent Annotation Dialogs for bounding box annotation. We train an agent to automatically choose a sequence of actions for a human annotator to produce a bounding box in a minimal amount of time. Specifically, we consider two actions: box verification, where the annotator verifies a box generated by an object detector, and manual box drawing. We explore two kinds of agents, one based on predicting the probability that a box will be positively verified, and the other based on reinforcement learning. We demonstrate that (1) our agents are able to learn efficient annotation strategies in several scenarios, automatically adapting to the difficulty of an input image, the desired quality of the boxes, the strength of the detector, and other factors; (2) in all scenarios the resulting annotation dialogs speed up annotation compared to manual box drawing alone and box verification alone, while also out- performing any fixed combination of verification and draw- ing in most scenarios; (3) in a realistic scenario where the detector is iteratively re-trained, our agents evolve a series of strategies that reflect the shifting trade-off between verification and drawing as the detector grows stronger.

  • Learning Intelligent Dialogs for Bounding Box Annotation.
    K. Konyushkova, J. Uijlings, C. Lampert, V. Ferrari. Spotlight at CVPR 2018.
    PDF code

Learning Active Learning

Active learning for image segmentation

Learning active learning from real and synthetic data

We suggest a novel data-driven approach to active learning (AL). The key idea is to train a regressor that predicts the expected error reduction for a candidate sample in a particular learning state. By formulating the query selection procedure as a regression problem we are not restricted to working with existing AL heuristics; instead, we learn strategies based on experience from previous AL outcomes. We show that a strategy can be learnt either from simple synthetic 2D datasets or from a subset of domain-specific data. Our method yields strategies that work well on real data from a wide range of domains.

  • Learning Active Learning from Data.
    K. Konyushkova, R. Sznitman, P. Fua. NIPS 2017.
    PDF poster code

How to learn to segment images without laborious annotation?

Active learning for image segmentation

Active learning for image segmentation

We propose an active learning (AL) approach to training a segmentation classifier that exploits geometric priors to streamline the annotation process in 3D image volumes. To this end, we use these priors not only to select voxels most in need of annotation but to guarantee that they lie on 2D planar patch, which makes it much easier to annotate than if they were randomly distributed in the volume. The geometry-inspired AL algorithm is also effective in natural 2D images. Moreover, an extended version of algorithm is developed for the multi-class segmentation problem. We evaluated our approach on binary and multi-class segmentation tasks in Electron Microscopy and Magnetic Resonance image volumes, as well as in natural images. It was demonstrated that the proposed AL approach is capable of dramatically decreasing the amount of tedious labelling for humans. Comparing our approach against several accepted baselines demonstrates a marked performance increase.

  • Introducing Geometry in Active Learning for Image Segmentation.
    K. Konyushkova, R. Sznitman, P. Fua. ICCV 2015.
    PDF video
  • Geometry in Active Learning for Binary and Multi-class Image Segmentation.
    K. Konyushkova, R. Sznitman, P. Fua.
    PDF

God(s) Know(s): analysing children drawings of God(s)

Analyzing children drawings of God

Developmental and cross-cultural patterns in children drawings of God(s)

We analyze the database of God (s) drawings by children from different counties, cultural backgrounds and ages. We detect developmental and cross-cultural patterns in children's drawings of God(s) and other supernatural agents. We develop methods to objectively evaluate our empirical observations of the drawings with respect to: (1) the gravity center, (2) the average intensities of the colors green and yellow, (3) the use of different colours (palette) and (4) the visual complexity of the drawings. We find statistically significant differences across ages and countries in the gravity centers and in the average intensities of colors. These findings support the hypotheses of the experts and raise new questions for further investigation.

  • God(s) Know(s): Developmental and Cross-Cultural Patterns in Children Drawings.
    K. Konyushkova, N. Arvanitopoulos, Z. Dandarova Robert, P.-Y. Brandt, S. Süsstrunk.
    PDF

Media coverage:

Find me what, don't know what

Analyzing children drawings of God

Exploratory search of scientific literature with intent modeling

One of the most challenging tasks of present-day scientists is making sense of the vast amount of scientific literature. In this project I worked in a team of researchers on SciNet, an exploratory information seeking system for scientific literature. SciNet redesigns the query-based search paradigm by combining interactive topical visualization of the information space with a reinforcement learning based user model and search engine. My role in the project was mostly in reinforcement learning user modeling and evaluation of retrieval performance. Through the interactive visualization, the system allows users to navigate in the information space by drilling down to more specific content or explore more general content. The system can assist users even when their goals are uncertain, underspecified, or changing. We evaluated SciNet with twenty scientists, comparing it to the popular query-based alternative Google Scholar. SciNet significantly improved the users' information seeking performance, in particular by offering more novel and diverse results while sustaining the overall quality of the results.

  • Directing exploratory search with interactive intent modeling.
    T. Ruotsalo, J. Peltonen, M. Eugster, D. Głowacka, K. Konyushkova, K. Athukorala, I. Kosunen, A. Reijonen, P. Myllymäki, G. Jacucci, S. Kaski. CIKM 2013.
    PDF
  • Supporting exploratory search tasks with interactive user modeling.
    T. Ruotsalo, K. Athukorala, D. Głowacka, K. Konyushkova, A. Oulasvirta, S. Kaipiainen, S. Kaski, G. Jacucci. AISTATS 2013.
    PDF
  • Scinet: A system for browsing scientific literature through keyword manipulation.
    D. Głowacka, T. Ruotsalo, K. Konyushkova, K. Athukorala, S. Kaski, G. Jacucci. IUI 2013.
    PDF
  • Bayesian Optimization in Interactive Scientific Search
    T. Ruotsalo, J. Peltonen, M. Eugster, D. Głowacka, K. Konyushkova, K. Athukorala, I. Kosunen, A. Reijonen, P. Myllymäki, G. Jacucci, S. Kaski. NIPS 2014 workshop on Bayesian Optimization.
    PDF

Analyzing children drawings of God

Exploratory image retrieval of vaguely defined targets

We consider a problem of image retrieval in a database of images without matadata such as keywords, tags or natural language text. In the situation when the user is unable to query the system by context, we employ a mechanism for relevance feedback to specify the information needs. For instance, imagine a journalist looking for an illustration to his or her article about loneliness in the database of unannotated photos. The idea of a suitable picture is very vague and the only opportunity for a journalist to navigate is to give relevance feedback to the images proposed by the system. The system operates through a sequence of rounds, when a set of k images is displayed and the user should indicate which one is the closest to their ideal target image.

  • Content-based image retrieval with hierarchical Gaussian Process bandits with self-organizing maps.
    K. Konyushkova, D. Głowacka ESANN 2013
    PDF
  • ImSe: Instant Interactive Image Retrieval System with Exploration/Exploitation trade-off.
    K. Konyushkova. M.Sc. thesis, 2013
    PDF video slides
  • ImSe: Instant Interactive Image Retrieval System with Exploration/Exploitation trade-off.
    K. Konyushkova, D. Głowacka
    Russian Summer School in Information Retrieval, 2014
    PDF