News

Our ultimate goal is to contribute to advancing AI from narrow to “broad” (general) while ensuring AI Safety

and alignment with human values, and contribute towards advances in other fields (healthcare, biomedical

sciences, and others) via developing generic, powerful large-scale models pretrained in a self-supervised

manner on broad variety of datasets. Such models can serve as a foundation of transferable knowledge

and can be used in a broad variety of applications ("downstream tasks") due to their drastically improved

generalization abilities as compared to prior state-of-art in the field of AI. More specifically, building on

recent successes in this area, we plan to train large-scale neural network models called Transformers, which

recently demonstrated impressive performance in language modeling and image processing; we plan to

evaluate their scaling with increasing model and pretraining dataset size, as well as the amount of compute

available. Such models, also known as "foundation models", appear to improve their generalization and

few-shot learning abilities with scale. We plan to investigate this trend in more detail, and identify the

most promising approaches to scaling the architecture, and datasets. Next, we plan to extend these models

to handle a much wider range of modalities beyond text and images, as well as various machine-learning

tasks, and expand them towards adaptive, continually learning systems. Finally, we plan to use the obtained

foundation models for predictive modeling in several applications such as healthcare and brain imaging. As

an outcome of this project, we will obtain highly transferable multi-modal models that we will make publicly

available. Hence we also see this project as a step towards the democratization of large multi-modal models

for the broader research community. The public availability of these models would allow researchers to

investigate both strengths and weaknesses of large multi-modal models, and further improve their transfer

capabilities to be used widely across various scientific domains.