This research study is studying a combination of drugs with radiation therapy as a possible treatment for Microsatellite Stable Colorectal Cancer, Pancreatic Cancer, or MSI High Colorectal Cancer. The interventions involved in this study are: * Nivolumab * Ipilimumab * Radiation Therapy

This research study is a Phase II clinical trial. Phase II clinical trials test the safety and effectiveness of an investigational intervention to learn whether the intervention works in treating a specific disease. "Investigational" means that the intervention is being studied. The FDA (the U.S. Food and Drug Administration) has not approved nivolumab for this specific disease but it has been approved for other uses. The FDA (the U.S. Food and Drug Administration) has not approved ipilimumab for this specific disease but it has been approved for other uses. Researchers hope to study the effects of the combination of Nivolumab and Ipilimumab. Many cancers use specific pathways (such as PD-1/PD-L1 and CTLA-4) to evade the body's immune system. Nivolumab and ipilimumab work by blocking the PD-1/PD-L1 and CTLA-4 pathways and thus releasing the brakes on the immune system so it can stop or slow cancer. Ipilimumab and Nivolumab are both antibodies. An antibody is a cell that attaches to other cells to fight off infection. The antibodies in ipilimumab work by not allowing cancer cell growth. The antibodies in nivolumab work by causing programmed cell death of the cancer cells. Radiation therapy is believed to increase the likelihood of response of immunotherapy (the prevention/treatment of a disease through an immune response). In this research study, the investigators are studying the combination of nivolumab, ipilimumab and radiation therapy on participants with microsatellite stable colorectal cancer, pancreatic cancer, or MSI high colorectal cancer. The combination of these study drugs have been tested and optimized for safety and is currently being tested in multiple disease types. The study drugs have not been tested and optimized in combination with radiation therapy. The investigators believe that through the combination of the study drugs and radiation therapy the body may produce an immune response to stop the cancer cells from growing.