Evaluating Candidates

The crux of the selection process is the evaluation of the candidates. Evaluating candidates can be extremely time-consuming so it's important to make good strategic and tactical choices on how to conduct the evaluation so you can achieve a good decision in the length of time you allotted to it. The goal of this section is to discuss how to gather all the information that you need in the shortest amount of time.

Keeping Track

Evaluations can be confusing, especially if you are evaluating lots of different vendors. Keeping good records avoids mistakes and saves time, and it doesn't need to take lots of resources. Record keeping is the project manager's responsibility. Since the end goal is to evaluate each aspect of the requirements matrix for each vendor, a good strategy is to summarize all the information in that matrix, keeping supporting arguments arranged by vendor and requirement number. Keep the records centralized so all team members can access them.

To RFP or not to RFP?

The traditional process for evaluating candidates is to use an RFP. The CRM project team prepares a detailed document (the RFP) that defines the tool or service being requested, what kind of information the vendor should provide, and how it should be packaged, and sends the RFP to all the vendors under consideration. The vendors prepare detailed responses to the RFP that are then evaluated by the organization. Based on the results of the evaluation the organization can create a short list of best-fit vendors. The RFP process is very useful in at least two ways. First, it forces the team to create a complete and detailed requirements list. If you are following the recommendations in this tutorial that won't be a problem but I cannot over-emphasize the benefits of knowing what you want before you go shopping. Second, the RFP process naturally creates a formal record of promises made by the vendors. The results of the RFP are usually attached to the sales contract and can be referred to later on if there is a problem with the deliverable. RFPs are almost always used for government contracts and very large contracts for that reason. If you want to use an RFP process, you can refer to the RFP template at the end of this chapter for inspiration. The template may also be useful if you are not planning to use an RFP: you can use some of the questions as a guide when meeting with the vendors. Despite its strengths, I find that the RFP process often fails to deliver the benefits one expects, for a number of reasons.

  • It takes an enormous amount of time. Creating the requirements list is something you should do anyway, so that's not so bad. But the level of detail and completeness you need to achieve to use the list in the RFP is much greater than what you would need for internal use, so it takes much more time than the requirements list itself.

    You also need to give the vendors several weeks to respond to the RFP—and if you give them too little time, the responses will not be as detailed and some of the vendors may decline to participate altogether. The answers also need to be carefully evaluated, including going back to the vendors for clarifications, which again takes weeks.

  • It is expensive. Tallying up the many hours that are required to create the RFP and to evaluate the answers may shock you. Price tags above $100k are not uncommon for the more complex proposals.
  • It can be misleading. The accuracy and commitment implied by RFPs seem to guarantee that they will provide correct answers from vendors, but it's not that simple. First, much of the value of the RFP lies in the quality of the requirements. Since CRM systems are complex, it's difficult to word each requirement accurately, succinctly, and unambiguously, and you can't expect vendors' answers to be any more precise than the requirements themselves. Second, CRM systems are very customizable, so vendors may state that a particular functionality is included when in fact it would require some level of customization. The promise of a complete and completely accurate response is not often met.
  • Many vendors actively discourage RFPs, especially those not at the high end of the market. It's always possible to insist on a response to an RFP, of course, but it requires some persistence and lots of time, which can be in short supply.

Because of the issues with the RFP process, it's often easier and better to use instead a streamlined process through which you actually walk through the requirements list and score each item yourself, based on your experience with each vendor's product. One way of thinking about the streamlined process is that it's very much like the RFP process except that you gather the responses yourself rather than having the vendor write them down before the evaluation. Using a streamlined process has many advantages.

  • It can be quite a bit faster than the RFP process, since you don't have to prepare a formal document or wait for the vendors to respond (or decipher cryptic answers!)
  • It is likely to yield higher-quality information. The probability that items will be completely misinterpreted by either the vendor or your RFP evaluation team is much less with a streamlined process. Also, the CRM team is more directly involved in the scoring and so will naturally focus on items of critical interest.
  • It is less resource-intensive. Not having to create the RFP document is a savings. Although the scoring requires much more work since you need to do it yourself, it's usually easier than evaluating RFP responses. You should find that the streamlined process requires fewer resources, at least if you are shooting for a high-quality evaluation.
  • It allows you to flexibly revise the requirements list during the evaluation process as you discover new information. Clearly you should strive to have a complete and accurate requirements list right from the start, but it's not unusual to make changes to the requirements list during the evaluation. Managing changes in an RFP process is difficult and confusing.

I very much believe in using a streamlined process but it does have a couple of drawbacks. One, the RFP process forces you to create a very clear and complete list of requirements. If you decide to use a streamlined process instead but you are not disciplined enough to create a strong requirements list, you may end up with a very poor fit because you never bothered to define what a good fit would be. Another situation in which an RFP process might be preferable is if you need to attach the formal results of the evaluation to the contract. If you want to use your scoring sheet for that purpose you will need to get it approved by the vendor, which will pretty much negate the time savings of the streamlined process. However, you will still benefit from the higher quality of information you obtain by doing the scoring yourself. Seriously consider using a streamlined process over a standard RFP.

Setting Up Productive Vendor Meetings

Regardless of whether you use an RFP or a streamlined evaluation method, being able to orchestrate productive interactions with vendors will go a long way in shortening the evaluation cycle and, more importantly, in ensuring that you gather accurate and complete information. This important coordination work is usually performed by the project manager, although some delegation is appropriate, at least for larger projects. To create the long list you might have used conference exhibits and webinars to get a feel for the various tools and vendors. During the evaluation you will want to switch to personal meetings that require coordination and preparation. For high-end tools, the meetings will most probably be handled face-to-face. For the lower-end tools some or all meetings may be held through web meetings and conference calls. The contents and attendance issues are very much the same regardless of the channel, however. The number and the organization of the vendor meetings depend on the complexity of the project, the size of the team, and logistical considerations. A common approach is to start with an initial meeting of the whole team (perhaps preceded by a smaller-scope evaluation from a smaller group) followed by a series of in-depth meetings attended by only those team members who are interested in the specific area being discussed. For instance, the business users (business owners and super-users) will want to see detailed demos of the functionality. See Screenshot for an overview of the meetings. The balance of this section describes recommended audiences and agendas and specifically describes how to get better demos.

Screenshot Vendor Meetings

Java graphics 06fig01.gif

Who Should Participate?

It's very important to hold most vendor meetings with every team member in attendance. It's sometimes desirable or necessary to hold focused meetings for only a part of the team (for instance, an in-depth architectural discussion with only the technical staffers). However, holding separate meetings as a matter of course makes for a disjointed and longer evaluation process, and eventually a poorer decision as each component is evaluated independently of each other. The project manager must orchestrate the various meetings. The process starts with the introductory meeting, a short meeting with the full team, branching off as needed into specialized meetings, and culminating in a longer group meeting for the final evaluation. The final meeting is almost always held face-to-face, often at the vendor's headquarter so all key staff members on the vendor's side can be present. Although it's certainly possible to include some team members through teleconferencing while others are physically in the same room, I find that such meetings quickly focus on the participants who are in the room and ignore the remote participants. It's very difficult to stay involved when one cannot hear all the conversation, cannot see what's being demonstrated, or cannot participate easily in the conversation. If you must hold meetings with some but not all participants on a teleconference, take great pains to ensure active participation from the remote participants. Send them materials ahead of time so they can preparefor the meeting and follow along during the discussion. Test the web conferencing connections and other tools that will be used during the discussion. Make a point to include remote participants in the discussions. If your project team is very large, it makes sense to have a core group do an initial evaluation of the vendors, bringing the entire team into the process only for vendors who pass the initial evaluation. If you choose to use a core group:

  • Make sure all functions are represented in it. A core group composed entirely of super-users will miss technical architecture items, and a group of technical staffers will miss business functionality.
  • Include different hierarchical levels in the core group. Don't use the core group idea to shield the business owners and the IT owner (or to shield the super-users and the technical staffers). To ensure the quality of the initial evaluation all levels must be included in the core group.
  • Try to use the same core group to do all the initial evaluations. This means that it will have the same blind spots (bad) but it also means that it will get more efficient with each evaluation (good) while keeping a good level of consistency (also good).
  • Don't use the core group for the entire process, only for the initial evaluation. If you feel the core group can indeed handle the whole thing, then perhaps your project team is too large, or not committed enough to the project.

You may think that there is a tremendous burden placed on the project team at this point. Multiple meetings with all the vendors on the long list? That's not often the case. Many vendors can be eliminated after the initial meeting, at least if you use the initial meeting to focus on the critical requirements. Especially with a large project team or a long list with many vendors, a suitable core group should be able to handle the initial evaluation so that the entire team only has to meet multiple times with a handful of vendors.

What Should Be on the Agenda?

If you let the vendor set the agenda for the initial meeting, you can expect a rather lengthy presentation on the company, the CRM field, and the high-level product architecture, followed by a brief and slick demo highlighting the more clever features of the product. The demo is usually customized to your requirements to the extent that they are known by the sales rep. A "standard" initial meeting is slick and smooth. It also makes for a terrible way to evaluate the product. Since standard vendor presentations don't work, you should always set the agenda for vendor meetings. Define both the topics you are interested in and how much time to devote to each. Remember the five categories of requirements?

  • Vendor
  • Technical architecture
  • Functionality
  • Implementation and maintenance
  • Price

You should aim to cover all five categories in the initial meeting. Of course, you shouldn't bother to dig very much into implementation and maintenance topics until you are satisfied that the tool can deliver the functionality that you need, and you can't expect to get a final quote that early in the process. A typical first meeting can last an hour to an hour and a half (more is better if you want a meaningful demo) and can be structured as follows:

  • Vendor introduction— no more than 10 minutes (cut off the vendor after 10 minutes to ensure that you have enough time for the rest).
  • Functionality— 5 minutes for a short presentation and 20 minutes for a demo. Since twenty minutes is not much for a demo, this is where to invest additional time if you can stretch the meeting to the recommended 90 minutes. Otherwise, plan a follow-up meeting if you like what you see the first time around. We'll talk more about demos in the next section.
  • Technical architecture— 10 minutes. This topic will bore the business users and cannot be covered adequately in less than an hour anyway, so you will need a follow-up meeting for the technical staffers but address essential concerns to be able to determine whether it's worth proceeding to the next step.
  • Implementation, maintenance, price— 5 minutes.
  • Q & A— 10 minutes.

The project manager needs to spend time with the vendor to prepare for the meeting and to ensure that the vendor fully understands that deviations from the agenda are not acceptable. Otherwise you will find yourself veering back to the standard blend of 95% PowerPoint and 5% canned demo, a most unsatisfying use of the team's time. The project manager also needs to assertively redirect meetings that do not follow the requirements. Better set the stage in the first meeting than suffer through unproductive presentations. The project manager should conduct a feedback session with the team shortly after the initial meeting. If the meeting is held at the vendor's site, it can be difficult to convene right after the presentation, but try to hold the debriefing very quickly afterwards. The goal of the debriefing is to decide whether to continue evaluating the vendor, as well as to improve future meetings and presentations. Short debriefings should be conducted after each meeting. If only a core group participates in the initial meeting, assuming that they like what they see, repeat the meeting with the entire team in attendance. You can take advantage of the repeat to further refine the agenda to meet your needs. After a successful initial meeting with the entire team, you will want to set up follow-up meetings to discuss specific areas. Plan on several follow-up meetings, targeting functionality, architecture, and implementation. The follow-up meetings can be attended by only a subset of the team members. However, it's always desirable to have at least one representative from a non-targeted group in attendance as it helps with information sharing and group decision-making. For instance, ask a super-user to attend the architecture discussion even though it's mostly for the technical staffers. A good format, if you can arrange it, is to schedule parallel meetings for the various subteams and to allow individuals to move from one track to the other for full coverage. Regardless of how many meetings are scheduled and how they are arranged, they must be planned in advance by the project manager or a team member to ensure that the agenda and attendees are appropriate. One of the main complaints of CRM team members is that they waste time in unproductive meetings. There will be plenty of irritants down the road over which you have little control, so make sure meetings are well organized. Use the outcome of the debriefing sessions to make each meeting more productive than the last one.

Getting a Meaningful Demo

Although demos are invaluable to share what the tool can and cannot do, many demos are poorly planned and end up being either divorced from (your) reality or excruciatingly dull as each field and each screen is visited and commented on without an overall vision of how real tasks can be performed. Here are practical ways to turn demos into useful selection tools.

  • Request a vanilla demo. It's fine if the vendor wants to put your logo on the screen to personalize the tool, but stay away from extensive customizations that obscure the actual implementation requirements. Generally speaking, the more time you give the vendor to prepare the demo the more customizations may creep in, so push for a reasonably close date and make it clear that you do not expect customizations.
  • Define what you want to see during the demo. An excellent strategy is to go through a normal workflow such as working a customer lead from inception to purchase, or working a customer service case from beginning to end. Stress that you are not interested in seeing each screen sequentially and in detail (you can always do that on the second pass if appropriate), but rather that you want to experience the flow of the product from one task to another.
  • During the demo, ask questions. If the demo shows corporate customers and your customers are consumers, ask how they can be accommodated. If the routing of service calls is by product and you want to route by geography, ask how you can be able to change that. If your normal workflow requires a manager's approval before sending a quote, ask how that can be built in.
  • As much as possible, drive the demo yourself. This may not be appropriate on the very first go-around, as you are still getting oriented to the product, but you should definitely do it later on in the process. If the tool is very new, driving the demo allows you to uncover what I politely call soft spots—plainly said, features that do not work yet. Regardless of the maturity of the tool, driving the demo gives you a very real feel for how intuitive it is. Experienced demo givers make it look effortless to use the tool, and only when you are doing the typing do you realize that the screens are poorly laid out, or that the workflow is not what you need.
  • If possible, get an evaluation copy of the software to have complete freedom for testing the software. This is now standard with ASPs, since they are all set up for that, but still pretty rare for the licensed package vendors. Vendors are often reluctant to set up evaluations because they fear, often rightly, that evaluations delay the sales cycle while requiring lots of effort on their part. If you decide to use a hands-on evaluation, set a reasonably short schedule and create a formal project plan. Hands-on evaluations are very time-consuming, so eliminate less promising candidates before embarking on them.

    During a hands-on evaluation, ask the super-users to perform the tasks they would normally perform with the software, noting each area of discomfort as well as each hole in functionality. If you have a good workflow defined for the particular business function it should not be too hard to do. For instance, support reps should create cases, work cases, and close cases. Working in a customer role they should be able to use the support portal to open cases, check status, and add to existing cases.

    Meanwhile, the technical staffers can put the software through its technical paces. Is it fast enough? Is it stable enough? Can customizations be accomplished easily?

    Often the evaluation copy will be installed at your site by the vendor. Assign appropriate technical staff to be present during the installation to see how difficult it is. In addition, the vendor staffers that are doing the installation are quite open and much can be learned through informal conversations with them. The smart project manager may want to hang out in the data center on the day of the installation and chat them up.

Tough Questions for CRM Vendors

Cleverly arranged demos are very powerful, and so are pointed questions to the vendor. I list here questions about the technical and functional aspects of the product. Pricing-specific questions are covered in , "Buying CRM Systems," and questions about integrators are covered in . For best effect, ask the tough questions several times during the evaluation from different individuals and compare the responses you get for consistency.

  • Are you using your own tools in-house?

    All vendors have customers. If the vendor is not using its own tools in-house, something's very fishy.

    Ask for a demo of the way the app is used in-house. Does it look anything like what you saw in the demo? Usually the in-house version is kept pretty close to the vanilla version, both because it makes it easier for upgrades and also because the cobbler's children are poorly shod. Question any differences between the customer demo and the in-house demo. The demo giver is often a regular employee, not a sales rep, so you have another opportunity to get candid input.

    Is the version used the current version? If not, why not? Using older versions internally is a sign that upgrades are painful. Make a note of it.

    Is the tool integrated with any other tools within the organization? If not, it may be yet another manifestation of the cobbler's children getting poor footwear, or it could be that integrations are complex and difficult. Try to find out why.

    Ask how many people are responsible for administering the system and what their skills are. Also find out how many users the system supports and what problems they are encountering with it.

  • Is it vanilla or is it custom?

    This question should be repeated for each and every feature of the product. A most useful question during demos, it can also be refined to "What work is required to create this particular customization?" Some tools have easy-to-use facilities to create minor customizations, so that minor changes are not an issue.

  • What happens to customizations when you upgrade?

    Upgrades are often very difficult for CRM tools because customizations need to be examined one by one to decide whether they are still needed for the new version. Then, the ones that need to remain need to be re-implemented against the new version. Grill the vendor on this topic as much as possible and request recommendations for creating customizations that minimize the maintenance requirements in the future. Questions about how to handle customizations during upgrades should be on your list when you talk to references as well.

    Another good question about upgrades is to ask the percentage of customers who are running the latest release. Unfortunately, there's no way to audit the answer, but the number of customers running the new release is a good indication of how difficult it is to upgrade versus how compelling new features are.

  • Do you have a user group?

    A vendor without a user group would be rather suspect to me. Ask whether the user group is driven by the vendor or is independent. Most user groups receive significant financial and operational assistance from the vendor, which is certainly a good investment in terms of customer satisfaction and marketing.

    If there is a user group, take time to talk to its representatives, keeping in mind that often the people who are active in the user group are great fans of the product.

  • Can we talk to your support manager?

    Speaking as an ex-support manager, I know that all the dirty laundry eventually gets aired to that group. While the support manager will be restrained when talking with a prospect, much interesting information can be gleaned from seemingly innocent questions. Ask how many support staffers are in place today, what kind of profile they have, and what the current backlog of cases is compared to the incoming volume. (Remember that CRM systems are complex, so a backlog of about two weeks worth of cases is normal—but much more may mean that problems are hard to troubleshoot). You should also ask what percentage of cases are bug-related (ask this question of multiple individuals—more than 10% indicates that the product is overly buggy).

  • What key features are you planning to release over the next year?

    While you must refrain from buying a system based on future features, which may or may not be released, and may or may not be released on schedule, it's very useful to understand the short-term product priorities. Do they fit with your priorities? If not, then perhaps another vendor would provide a better fit.

Another source of relevant questions is the RFP template at the end of this chapter. The questions are valid whether or not you are planning to use a formal RFP process.