Justice Brown and his wife, Cindy, have been married over thirty-nine years. They have three children, Joshua, Caroline, and Emily, and a daughter-in-law, Jennifer. They also have four grandchildren.
Justice Brown is an active member of First Baptist Church of Houston where he teaches a young marrieds class. He coached soccer, basketball, baseball, and softball for over a dozen years for his three children and basketball for his grandson. He was the president of the Houston chapter of the Christian Legal Society for a number of years.
He currently serves as the past board chair of the board for KSBJ radio station in Houston after serving as its vice chair and then chair for six years. For the past three years he has served as the secretary of the board of Spring Branch Family Development Center, an organization that provides enrichment and support services to a historically under-served population. He is also a volunteer for Good Stewart Global Initiative, which builds libraries in Africa. He served for two years as the chairperson of the Memorial High School Community Network, a coalition of parents, teachers, and administrators with a common objective–to foster a school and community environment that encourages teenagers to choose and follow a healthy, alcohol-free and drug-free lifestyle. He is also a former board member of Houston Habitat for Humanity; he served as its vice-president when it built one hundred affordable homes in one week and later as its president.
He has been involved in a number of pro bono cases including acting as lead counsel in a successful challenge to a school district policy that barred high school students from holding student-initiated and student-led Bible studies on campus after school. That case was the first in a series of cases defending the constitutionality of the Equal Access Act, which required federally-funded secondary schools to provide equal access to their facilities for all extracurricular student groups.