This trial studies the side effects and how well high-dose brachytherapy works in treating patients with prostate cancer that has not spread to other parts of the body. Brachytherapy is a type of radiation therapy in which radioactive material sealed in needles, seeds, wires, or catheters is placed directly into or near a tumor and may be a better treatment in patients with prostate cancer.

PRIMARY OBJECTIVES: To estimate the rate of acute (within 6 months of high-dose rate \[HDR\] completion) grade ≥ 2 genitourinary (GU) toxicity following high-dose-rate (HDR) brachytherapy (BT) as monotherapy for newly-diagnosed prostate cancer using the National Cancer Institute's Common Terminology Criteria for Adverse Events, version 3 (CTCAE v3.0). SECONDARY OBJECTIVES: * Estimate the proportion of men with a prostate-specific antigen (PSA) nadir by one year (nPSA12) of \< 2 ng/mL. * Estimate the rate of freedom from biochemical failure at 5 years (FFBF). * Evaluate patient-reported quality of life via the 32-item Expanded Prostate Cancer Index Composite (EPIC). * Assess the cost-effectiveness of HDR BT as monotherapy for prostate cancer using the 6-item European Quality of Life 5-Dimensions (EQ-5D). * Explore pre-treatment clinical risk factors to optimize patient selection for HDR BT as monotherapy for prostate cancer. * Compare acute and late (\> 6 months after HDR completion) GU and gastrointestinal (GI) grade ≥ 2 toxicity using CTCAE v3.0 and v4.0. * Explore dosimetric predictors of toxicity. Patients undergo high-dose-rate brachytherapy over 2 fractions. Patients may receive androgen deprivation therapy (ADT) comprising bicalutamide orally (PO) once daily (QD). Patients may also receive luteinizing hormone-releasing hormone (LHRH) agonist therapy comprising leuprolide acetate intramuscularly (IM) or subcutaneously (SC), goserelin acetate SC, triptorelin pamoate IM, or degarelix SC for 4 to 6 months (intermediate-risk patients receiving ADT) or 6 to 36 months (high-risk patients) at the discretion of the treating physician. After completion of study treatment, patients are followed up at 3, 6, 9, and 12 months, and then yearly for up to 5 years.