Quick eval of Gemini 3 dev tools
As often with Google - great AI, underwhelming product tools
Maybe Gemini 3 is the best model right now, maybe it’s not, I still don’t know.. Last night I tried to give it a shot, check it out, time-boxing the whole exercise to 15 minutes, and unfortunately, as is often the case, Google’s product tooling implementation was the weak link and hurt the overall impression.
Task
Iterate on a very simple weather MCP server, which hits National Weather Service API to get weather prediction for a specific city, so I can objectively compare Gemini 3 vs Copilot with GPT-5.1, Codex, and Claude Sonnet 4.5, which I use daily.
Attempt 1: Gemini CLI (on my dev laptop)
It took a couple of minutes to get Gemini CLI set up and running (good!).
After installation it told me I could select models using the /model command, but in order to enable Gemini 3 Pro in a preview mode I needed to go to /settings and enable Preview features.
Ok, done.
After restarting Gemini CLI, I asked it to look at the project (a single .py file with an MCP server, about 100 lines of code), read the file, and write its findings into a markdown file in the local directory for future reference.
After thinking for about a minute, the answer: “You don’t have access to Gemini 3.”
What’s more, the URL printed in the second line of the message is not selectable — I can’t just copy and paste it into a browser, instead I have to type it in manually to find out why I don’t have access to Gemini 3.
Attempt 2: PyCharm plugin – Gemini Code Assist (by Google)
After installing the plugin and authenticating with my GCP account, I asked Gemini to make a small enhancement to my weather MCP server implementation - expose a new tool to provide multi-day weather forecast for a given location in a single request.
It sat there spinning for about 5 minutes, so I had to kill it.
Another issue - there is no way to select — or even see — which model the plugin is using, was it Gemini 3 preview, or 2.5 Pro, or..? 🙁
To be fair- after killing and restarting PyCharm - Gemini plugin worked on the second attempt, and even implemented the new MCP tool to work exactly as I prompted.
However, there is still no way to see or configure which models Google Code Assist plugin is even using.
Upon closer examination of the gemini-backend.log file in the PyCharm logs directory i saw this:
“modelConfig”:{”displayName”:”2.5 Flash”,”id”:”chat-gemini-2-5-flash-free-tier”}
Again, there is no UI / settings page within the plugin on IDE settings to see or change which model is used 🙁.
By this point 15 minutes I allocated for this eval project expired..
Conclusion
Very underwhelming first experience.
Google still has a lot of work to do on its user-facing developer tools, both the Gemini CLI and the IDE plugin. They may have excellent models, but they continue to lose users because of poor UI and tooling.
By contrast, the Claude Code plugin for PyCharm works like a charm (pun intended)! Not a single issue, very easy to select which model I want to use, very intuitive interface, and configuring and testing the same reference MCP server with Claude Desktop took about 20 seconds!