To determine the feasibility of comparing the Childhood Arthritis and Rheumatology Research Alliance (CARRA) consensus treatment plans (CTPs) in treating new-onset, moderate juvenile dermatomyositis (JDM) using the CARRA registry, and to establish appropriate analytic methods to control for confounding by indication and missing data.A pilot cohort of 39 JDM patients from the CARRA registry was studied. Patients were assigned by the treating physician, considering patient/family preferences, to one of three CTPs: methotrexate and prednisone (MP), intravenous methylprednisolone, methotrexate and prednisone (MMP) or intravenous methylprednisolone, methotrexate, prednisone and intravenous immunoglobulin (MMPI) .The primary outcome was the proportion of patients achieving moderate improvement at 6 months under each CTP. Statistical methods including multiple imputation and inverse probability of treatment weighting were used to handle missing data and confounding by indication.Patients received MP (n=13), MMP (n=18) and MMPI (n=8). Patients in all CTPs had significant improvement in disease activity. Of the 36 patients who remained in the pilot study at 6 months, 16 (44%) of them successfully achieved moderate improvement at 6 months (6/13, 46% for MP; 7/15, 47% for MMP; 3/8, 38% for MMPI).After correcting for confounding there were no statistically significant pairwise differences between the CTPs (p = 0.328-0.88).We gained valuable experience and insight from the pilot study to guide the design and analysis of comparative effectiveness studies using the CARRA registry CTP approach. Our analytical methods can be adopted for future comparative effectiveness studies and applied to other rare disease observational studies.