Do not speak evil against one another, brothers. The one who speaks against a brother or judges his brother, speaks evil against the law and judges the law. But if you judge the law, you are not a doer of the law but a judge. There is only one lawgiver and judge, he who is able to save and to destroy. But who are you to judge your neighbor? - James 4: 11-12 (ESV)
There's a little more here than merely don't judge your neighbor. The context reveals that James is clearly dealing with speaking evil against your neighbor. Other translations refer to this as slander. Commentaries agree that this is when someone is providing false reports or false testimonies about someone. So again if your point is that we should not provide false witness against our neighbors - I wholeheartedly agree.
The sadly ironic thing is that we get things so backwards in the church. We are constantly judging the world. They are sinners. They are heathens. They are going to hell. Look at what they are doing to this country. They have taken prayer out of schools and the Ten Commandments out of courthouses. Christians then hold up signs in protest, telling them God hates them. Then an avowed Christian falls into sin and the collective cry is - Judge not! Let us set the record straight biblically:
For what have I to do with judging outsiders? Is it not those inside the church whom you are to judge? God judges those outside. "Purge the evil person from among you." - 1Corinthians 5: 12-13 (ESV)
Those in the world are facing the judgment of God. We are not supposed to shout about their sins but rather shout about He who can save them from their sins. On the other side, we are supposed to deal directly with the sin in our own camp. If we dig further into the context of 1Corinthians 5 we will see that the church at Corinth was turning a blind eye to someone who was committing sexual sin with the wife of their father.
It is actually reported that there is sexual immorality among you, and of a kind that is not tolerated even among pagans, for a man has his father's wife. And you are arrogant! Ought you not rather to mourn? Let him who has done this be removed from among you. - 1Corinthians 5: 1-2 (ESV)
This is what we do today in the church. Maybe not the same exact sin but the same exact response. Blind pious arrogance in the face of what God hates. Let us be honest beloved. We simply do not mourn over our sin at all. We pay lip service to it. We seek out churches that will refuse to even talk about it. And if it happens to be pointed out by anyone we scream - judge not! Let us now exposit the whole context of the judge not verses to see exactly what Jesus is trying to say to us. Working backwards in the key verses we begin with the core revelation we all need to grasp:
Why do you see the speck that is in your brother's eye, but do not notice the log that is in your own eye? Or how can you say to your brother, 'Let me take the speck out of your eye,' when there is the log in your own eye? You hypocrite, first take the log out of your own eye, and then you will see clearly to take the speck out of your brother's eye.