Existing datasets for training pedestrian detectors in images suffer from limited appearance and pose
variation. The most challenging scenarios are rarely included because they are too difficult to capture due
to safety reasons, or they are very unlikely to happen. The strict safety requirements in assisted and
autonomous driving applications call for an extra high detection accuracy also in these rare situations.
Having the ability to generate people images in arbitrary poses, with arbitrary appearances and embedded in
different background scenes with varying illumination and weather conditions, is a crucial component for the
development and testing of such applications. The contributions of this paper are three-fold. First, we
describe an augmentation method for controlled synthesis of urban scenes containing people, thus producing
rare or never-seen situations. This is achieved with a data generator (called DummyNet) with disentangled
control of the pose, the appearance, and the target background scene. Second, the proposed generator relies
on novel network architecture and associated loss that takes into account the segmentation of the foreground
person and its composition into the background scene. Finally, we demonstrate that the data generated by our
DummyNet improve performance of several existing person detectors across various datasets as well as in
challenging situations, such as night-time conditions, where only a limited amount of training data is
available. In the setup with only day-time data available, we improve the night-time detector by 17%
log-average miss rate over the detector trained with the day-time data only.