This phase I/II trial studies the side effects and best dose of ipilimumab and nivolumab when given together with brentuximab vedotin, and how well they work in treating patients with Hodgkin lymphoma that has returned after a period of improvement (recurrent) or has not responded to previous treatment (refractory). Immunotherapy with monoclonal antibodies, such as ipilimumab and nivolumab, may help the body's immune system attack the cancer, and may interfere with the ability of cancer cells to grow and spread. Brentuximab vedotin is a monoclonal antibody, brentuximab, linked to a toxic agent called vedotin. Brentuximab attaches to CD30 positive cancer cells in a targeted way and delivers vedotin to kill them. It is not known whether giving brentuximab vedotin and nivolumab with or without ipilimumab may kill more cancer cells.

PRIMARY OBJECTIVES: I. To determine the maximum tolerated dose (MTD) and dose limiting toxicities (DLT) of the combinations of brentuximab vedotin and ipilimumab, brentuximab vedotin and nivolumab, and brentuximab vedotin, ipilimumab, and nivolumab. (Phase I) II. To evaluate the complete response (CR) rate for the regimens of brentuximab vedotin and nivolumab compared to brentuximab vedotin, ipilimumab, and nivolumab. (Phase II; adult cohort \[aged \>= 18 years\]) III. To characterize the safety and toxicity of treatment combination in the pediatric population. (Phase II; pediatric cohort \[aged 12-17 years\]) SECONDARY OBJECTIVES: I. To evaluate complete response (CR) rate, partial response (PR) rate and overall response rate (ORR), for the combinations of brentuximab vedotin and ipilimumab, brentuximab vedotin and nivolumab, and brentuximab vedotin, ipilimumab, and nivolumab. (Phase I) II. To evaluate the duration of remission (DOR) to these combinations and compare with the DOR achieved with the most recent prior systemic therapy. (Phase I) III. To evaluate the progression-free survival (PFS) and the overall survival (OS) in patients receiving the combination of brentuximab vedotin and ipilimumab, brentuximab vedotin and nivolumab, and brentuximab vedotin, ipilimumab, and nivolumab. (Phase I) IV. To evaluate the ORR, PR, and stable disease (SD) rate for the combinations of brentuximab vedotin and nivolumab and brentuximab vedotin, ipilimumab, and nivolumab. (Phase II) V. To evaluate the DOR to these combinations and compare with the DOR achieved with the most recent prior systemic therapy. (Phase II) VI. To evaluate the 5 year PFS and OS in patients receiving the combinations of brentuximab vedotin and nivolumab and brentuximab vedotin, ipilimumab, and nivolumab. (Phase II) VII. To further evaluate the safety and characterize the toxicity for the combinations of brentuximab vedotin and nivolumab, and brentuximab vedotin, ipilimumab, and nivolumab. (Phase II) CORRELATIVE STUDY OBJECTIVES: I. To evaluate the ability of these combinations to alter tumor specific T cell immunity. (Phase I) II. To evaluate the effects of these combinations on systemic immunity. (Phase I) III. To evaluate a panel of cytokine and T cell specific biomarkers from the peripheral blood as a potential immune signature of treatment response to therapy with these combinations for patients with relapsed/refractory Hodgkin lymphoma (HL). (Phase I) IV. To evaluate using gene expression profiling (GEP) a signature of response to these novel combinations of an antibody drug conjugate with immunomodulatory therapy. (Phase I) V. To evaluate the ability of these combinations to alter tumor specific T cell immunity, and circulating T cell phenotypes, in patients as a function of treatment response at multiple timepoints during therapy. (Phase II) VI. To evaluate peripheral blood cytokine profiles in responding and resistant patients at multiple timepoints during therapy. (Phase II) VII. To evaluate using GEP a signature of response versus (vs.) resistance to these novel combinations of an antibody drug conjugate with immunomodulatory therapy. (Phase II) VIII. To evaluate the influence of human gut microbiome dysbiosis on HL lymphomagenesis and the systemic immune response. (Phase II) IMAGING CORRELATIVE STUDY OBJECTIVES: I. To evaluate atypical response patterns with currently available response evaluation criteria. (Phase II) II. To correlate response evaluated using currently available response evaluation criteria with duration of response (PFS, event free survival \[EFS\], failure free survival \[FFS\]). (Phase II) III. To evaluate response patterns in different immunotherapy treatment schemes and correlate with historical data using chemotherapy. (Phase II) IV. To correlate imaging changes in all treatment schemes quantitatively with PFS. (Phase II) EXPLORATORY OBJECTIVES: I. Evaluate outcomes (CR, PFS) between patients with/without prior transplants. (Phase II) II. Evaluate outcomes (PFS, OS) between the patients who stay on treatment and do not go to transplant in both arms (the post auto and the few others who don't want transplant) vs the patients who go off for transplant. (Phase II) III. Evaluate outcomes (CR, PFS) in pediatric population (age 12 to \< 18 years of age) vs. adult population. (Phase II) OUTLINE: This is a phase I, dose-escalation study of brentuximab vedotin, ipilimumab, and nivolumab followed by a phase II study. PHASE I: Patients are assigned into 1 of 3 arms. ARM I: Patients receive brentuximab vedotin intravenously (IV) over 90 minutes on day 1 of cycles 1-16 and ipilimumab IV over 30 minutes on day 1 of cycles 1-4, 8, 12, and 16. Treatment repeats every 21 days for up to 16 cycles in the absence of disease progression or unacceptable toxicity. ARM II: Patients receive brentuximab vedotin IV over 90 minutes on day 1 of cycles 1-16 and nivolumab IV over 30 minutes on day 1 of cycles 1-46. Treatment repeats every 21 days for up to 16 cycles and every 14 days beginning cycle 17 for up to 46 cycles in the absence of disease progression or unacceptable toxicity. ARM III: Patients receive brentuximab vedotin IV over 90 minutes on day 1 of cycles 1-16, nivolumab IV over 30 minutes on day 1 of cycles 1-46, and ipilimumab IV over 30 minutes on day 1 every 12 weeks for up to 9 doses. Treatment repeats every 21 days for up to 16 cycles and every 14 days beginning cycle 17 for up to 46 cycles in the absence of disease progression or unacceptable toxicity. PHASE II: Patients are randomized to 1 of 2 arms. ARM I: Patients receive brentuximab vedotin IV over 30 minutes on day 1 of cycles 1-16 and nivolumab IV over 90 minutes on day 1 of cycles 1-34. Treatment repeats every 21 days for up to 34 cycles in the absence of disease progression or unacceptable toxicity. ARM II: Patients receive brentuximab vedotin IV over 90 minutes on day 1 of cycles 1-16, nivolumab IV over 30 minutes on day 1 of cycles 1-34, and ipilimumab IV over 30 minutes on day 1 every 12 weeks for up to 9 doses. Treatment repeats every 21 days for up to 34 cycles in the absence of disease progression or unacceptable toxicity. All patients also undergo computed tomography (CT) or positron emission tomography (PET) scan throughout the trial. Patients undergo blood sample collection and may undergo tumor biopsy on study. After completion of phase I study treatment, patients are followed up every 3 months for 1 year, then every 6 months for 2 years. After completion of phase II study treatment, patients are followed up for 10 years.