Checking References

Checking references is a big part of business life. We check references when we hire someone and we check references when we make a large purchase. I'm a firm believer that reference checks can give you additional insight that you could not have gotten otherwise, helping you make a better decision. But reference checks only work if they are conducted properly, and that's what is discussed in this section. Reference checks may be done to choose between two qualified vendors or to confirm that your favorite vendor is indeed solid. Often, extensive reference checks are done only for the favorite vendor. A reference check is just a check. It's not a journey of discovery, and it cannot replace the evaluation process. In fact, you can only check what you already know, or at least what you know enough to ask about. You can discover new facts during a reference check, but only if you are asking the right questions from the right people, and that requires proper preparation.

Why Do a Reference Check?

Reference checks take time and resources, especially if you want to get real information out of them. Why bother?

  • To confirm that what you learned through the evaluation is indeed correct. You don't need to check each and every detail, and the individual providing the reference check probably won't want to spend hours with you confirming each and every detail anyway. However, you should check that the critical functionality for you is indeed working properly. If you suspect trouble, then ask more detailed questions.
  • To extend your evaluation. CRM systems are complex beasts and there's simply no way that you can check everything yourself. For instance, the only practical way to check that it's possible to get the tool to talk to your accounting system is to ask someone who has done it successfully. Performance questions are also best answered through appropriate reference checking.
  • To get ideas. Although I will once again warn against rampant customizations, the fact is that references may inspire you to try wonderful ways to use the tool.
  • To get implementation tips. If you ask the right questions, of which we will see many examples shortly, you can and should find out plenty about what's required for a successful implementation including who should be involved, what time lines are reasonable, and hidden "gotchas." This kind of information is particularly difficult to get through other means, so make sure you focus part of the reference-checking questions on the implementation process.
  • Because it's required within your company process. Just because it's required doesn't mean it's useless bureaucracy. Gain from complying with the rules by making the most of your contact with the references.
  • To cover yourself. This may be inglorious but it's a sad reality that the one who does not check references gets blamed. My view on covering yourself is the same as for following the process, discussed above: if you have to do it, you might as well do it right and improve the quality of the evaluation.

When to Check References

Although this tutorial, by necessity, presents the tool selection process as linear, there is no reason to keep reference checking for the very end of the evaluation. Indeed, if you are working with a fledgling vendor or if you need confirmation on a specific point during the evaluation, it's not a bad idea to start checking references early. Don't expect to complete a reference check early in the process, however. The key to good reference checking is asking the right questions, as we will see later, so you can't be ready with all your questions until you have completed your own evaluation. It's fine to check references in stages, confirming some key points early on and coming back with a more detailed checklist later on. Always leave the door open for more questions if you need to.

Picking a Good Reference

The traditional way to get references is to ask the vendor to provide a list. There's nothing wrong with that, but of course the vendor will only give you good references (if not, it's time to end the romance), and may or may not give you accounts that are good fits for your particular requirements. To improve the odds of a good fit, specifically request accounts that match your requirements.

  • Accounts that have a similar number of users. This is about both scalability (can the tool support the user load) and, just as importantly, complexity (is the tool sophisticated enough to support the process complexity to match the number of users). If you have 500 users and you are interviewing a 40-user account, you simply won't get appropriate data on metrics or escalations, to name just two areas where complexity matter. Similarly, if you have 40 users but are talking to a 500-user reference, you may find that their elaborate customizations are completely inappropriate for your size.

    Don't be obsessed by matching exactly the number or the distribution of users as long as you are in the same ballpark. You don't need perfect clones.

  • Accounts that have similar business models. Look for organizations with similar customers rather than organizations in the same line of business. Actually, checking references with competitors is a nightmare since it creates competitive issues, so much so that the references may be unusable.

    Rather, concentrate on the reference account's relationship with their customers: are they selling to Fortune 500 companies or to consumers? Are they selling worldwide or only in the U.S.? Are they selling directly or through distributors? Do they sell standard goods or is each contract custom? Match the selling model rather than the specific field you are in.

  • Accounts that are in full production with the tool. It's not completely useless to talk to accounts that are in the implementation phase, but the proof is in the pudding so you absolutely must have at least a couple of references in production. This can be hard if you are working with a fledgling vendor but be warned that being at the forefront can be painful when it comes to CRM.

    While you should talk to customers that are in production, it's good to talk to at least one reference that implemented within the past year so you can get meaningful feedback on the implementation process.

  • Accounts that are using the same products or modules you are planning to use. There's no point to talk to a reference that's using the service-tracking module if you are planning to implement the sales-tracking module.
  • Accounts that are using the current release. Following up on the previous point, talk to accounts that are using the release you are planning to use. This can also be a challenge, especially with traditional vendors, since the upgrade process is difficult and established customers are in no rush to move up. Talking with accounts that are running the current release allows you to check that the new features you need are indeed working properly. If you are not that interested in new features, and assuming that the latest release is not a complete rewrite of the tool, you may relax this rule (and probe for how arduous the upgrades are).
  • Accounts that are using a similar technical infrastructure. If the vendor offers several choices of technical environments, or if you have specific architectural requirements, check that other customers are in production in similar environments. It's not absolutely crucial that the reference(s) with similar technical infrastructure also use the products in the way you will use them since this is more of a technical check. However, confirm that the user loads are similar.
  • Ideally, account that are close by so you can visit in person. A site visit may seem to be a crazy extravagance both in time and resources but if you can afford one you will learn a lot more than you could through a regular phone conversation. Site visits almost always feature a hands-on demo, which may be harder to stage from afar and which almost invariably will give interesting information. Site visits also allow much more contact with various users, allowing you to ask more questions and to get a more balanced view of usability and issues.

If the vendor mentioned some established accounts during the sales process that you think would be a good fit for your requirements, don't hesitate to specifically request to talk to them. It's not necessarily a bad sign if a particular account is not available for reference checking (after all, there's such a thing as reference-giving fatigue) but you might as well probe on those names that were flung out to impress you early in the evaluation process. There's nothing better than an unsolicited reference, so if you find people in your network who are using the tool, go talk to them even if they don't quite meet the criteria above. Unsolicited references are usually more open since they already have a relationship with you, even if it's slight, and unsolicited references are also more candid since they are not controlled by the vendor. Make allowances for the differences between the references' environments and yours when evaluating the answers. How many references should you talk to? Two or three references from perfect-fit accounts would make me very happy, and more would get repetitive, not to mention very time-consuming. Since you are unlikely to find many perfect fits, you may need to talk to slightly more than three, but each discussion will be shorter than with a perfect-fit account since you can focus only on one aspect of the tool. For instance, you may talk to one account about the implementation, to another about performance, and to yet another about the SFA features. Don't overdose on references. Strive for quality rather than quantity and remember that you have unique needs and a unique approach so what worked for others may not work for you, and what didn't work for others may work for you.

Whom to Ask

When checking references, you want to talk to at least two people, the business owner and the IT owner, so you can probe both the technical and the functional sides. Whenever possible, talk to end-users and system administrators for more details and, often, a more candid analysis of issues. The vendor will typically give you only one contact, which could be either the IT owner or a business owner. Simply ask that individual to give you the other contact during the reference check. It's usually a pretty simple matter.

Are References Impartial?

There's a great deal of appropriate concern about the level of impartiality of references. Vendors know very well, especially for enterprise apps, that positive references are a must for the vast majority of sales (in other words, most buyers aren't just dazzled by the cute demos). Vendors put a great deal of effort into cultivating references and the question is: do such efforts go too far? It's true that vendors often offer enticements to customers who are willing to take the time to serve as references. Part of it is completely normal: after all, serving as a reference takes time and resources, and it's quite appropriate to thank customers who have done a favor with a small trinket or a nice meal. However, you need to be aware that, especially in the high end, vendors may offer a variety of very tangible rewards to reference customers, including:

  • Product discounts
  • Special pricing on maintenance and support
  • Free training
  • Free attendance at user group meetings
  • The chance to participate in beta programs
  • A seat on the customers' advisory board (The customer advisory board, not to be confused with the corporate board, of course, nor with the user group, is a vehicle that many vendors use to collect formal input from customers on new releases and other strategic decisions. The existence of a customer advisory board is actually a good thing as it shows interest in getting customers' input into the planning process. Depending on the influence of the advisory board, members can hold quite a bit of power in getting features implemented that benefit them. Therefore, it's always interesting to note the composition of the board and the way board members are chosen. )
  • Access to top executives, either informally or formally. It's hard to put a value on this but it comes in handy when one needs to ask for a new feature or an expedited software fix.

Some larger vendors have a formal reference reward program in place that works very much like airline frequent-flier programs: participants get points as they give references and they can redeem the points as they wish. Is it a problem that references get rewarded? Not really, although you should find out what the rewards are, and weigh the feedback appropriately. While there is nothing wrong with a reward system, I feel that it should not be kept a secret. Ask the vendor and the references about a reward system even if it's not easy to get straight answers on this particular topic. Regardless of the existence of a formal reference reward program, you should be aware that reference accounts are likely to get preferential treatment from the vendor in a variety of ways, and in particular for support services. This means that you may experience less responsive support than what they describe during the reference check. Even if you have doubts about the impartiality of the references the vendor gives you, do check references! It's not unusual for buyers to simply rely on the long list of "big names" on the vendor's web site without ever checking further. This is a mistake. Listing a well-known Fortune 50 name as a satisfied customer may simply mean that some small subgroup in the company bought the product some time ago and may not be using it any longer. Check references and don't assume anything.

Who Should Do the Reference Check?

Typically the project manager checks references, but it's wonderful if the business owner(s) and the IT owner can join in, either as a group or individually as required to check out particular aspects. In any case, it's best if the entire project team can help create and approve the list of questions to ask the references. If an onsite visit is planned, take a cross-section of the team along, not just the business owners and the IT owner. Onsite visits create opportunities for end-users to talk to end-users and for technical staffers to talk to technical staffers in a much more open manner than standard reference checks. You will get much more out of the trip if you can prepare everyone to ask the right questions.

What to Ask

A reference check is only as good as the questions you ask. Most reference checks used on mediocre questions such as "How's the tool working for you?" and "Are you happy with the support you are getting from the vendor?" Vague questions beget vague answers, so you must be specific. The good news is that the questions are all ready for you, right in your requirements list. And as you work through the evaluation you should make a note of the points you want to confirm with the references, either because they are particularly crucial or because you can't quite pinpoint how the tool performs on them, so that by the time you check references you should already have a list of questions to ask. Part of checking references is simply to confirm that the information you got from the vendor is correct. So if the vendor tells you that the pre-packaged CTI integration kit makes for easy integration with brand X, you'll want to talk to somebody who did that particular integration and was successful with it. If the vendor says that upgrades are easy because customizations are clearly differentiated from the base code, talk to a customer who upgraded recently and find out how it went. As discussed above, you may need to check multiple references to confirm all your requirements. Reference checks must be prepared in advance. A thorough check (without a demo) may take a full hour and you should set aside a similar amount of time to prepare for it. After all, the person giving the reference is doing you a favor so you should minimize the interruption while ensuring your critical questions get answered. Customize the questions for each person you talk to: don't bother asking the long-term customer for detailed implementation stories, or asking the smaller account for scalability stories. However, it is very useful to ask some of the same questions from several references, as individual experiences vary. Here are 20 good questions to ask references.

  • How are you using the tool today?

    This is a general question to launch a discussion of the specific environment of the organization so you can check how well it matches your environment. In particular, you want to check: the user population including how many users there are, where they are located, and what their roles are; the business processes being managed through the tool; and the particular modules they are using.

    All these questions allow you to align the reference's environment to yours. If differences are large, the reference check may not be very valuable.

  • What is your technical environment?

    This question focuses on the technical architecture. Things to check include the exact products and versions they are using; the hardware and operating system for the hardware involved (servers, desktops, laptops, other devices you may require), the network setup, and the database management system.

    The IT owner should be able to answer these questions. Business owners usually don't have such a detailed view.

  • How long have you been in production?

    As mentioned above, a reference that's not yet in production is not a real reference, in my opinion. Make sure they've rolled out all the users, not just some small sample.

  • Were you a part of the team that decided to select this tool? What were the top reasons why you selected it? What other tools did you consider?

    If the reference is a long-term customer, which is not rare (and a good sign in and of itself!) it may be difficult to locate individuals who were on the original project team. Don't worry too much if you can't get to the original decision makers, especially for long-term customers, since the competitive landscape has probably changed a lot since the purchase. For more recent customers, however, it's always interesting to see what criteria they used and whether your findings match theirs.

  • Do you use this particular feature [that you are interested in]? What makes it valuable to you? What do you wish was different?

    This is where your evaluation of the tool comes in handy, both to focus on the truly important questions (you can't check all the features in a reference check) and to be able to interpret what the reference is saying. And this is where a demo may be required to truly make sense of what's possible, what's easy, and what may not work.

    Don't forget to ask about features of the development environment, not just the end-user environment!

  • What performance do you experience with the system? How quickly do screens repaint? How quickly do searches return results? Do you have performance concerns in any particular area? What if anything have you done to mitigate them?

    Performance can be a tricky area to discuss. First, there is the issue of comparable environments: don't bother asking performance questions of an account that's much smaller than you, for instance.

    Even if the environments are similar, what's good enough for a reference may not meet your requirements at all. Exchanging benchmark numbers may be helpful, but also consider a site visit or online demo for a visual confirmation. Also try talking to an end-user.

  • What customizations have you performed with the tool? Why?

    You are on the lookout for two things here: are there areas of the tool that require customizations you may need? And how painful are the customizations? Weigh the customizations that the reference account implemented against your own requirements. You may not need them.

    If a number of references tell you that they customized a particular area of the tool, it's likely that there is a weakness there and that you may need to also do work in that area. On the other hand, if none of the references touched a particular area that you feel needs work, pay attention. It can be a sign that either the references are not good matches for you, or that your process is outside the norm and should be changed rather than changing the tool.

  • Do you have a list of customizations you'd like to do in the future? How long is it? How old is the top-priority item on the list?

    The list of enhancement requests gives you clues about 1) whether the tool is really used and 2) how difficult it is to create customizations. If you find a reference that has no requests for enhancements, it could be that the tool is not used very much (not a good sign), or that the tool is perfect. Please drop me a note if you ever encounter the latter situation; I want to know! If the top-priority item is very old, it's likely that customizations are difficult.

  • Tell me about the implementation process for the tool. Were you a part of it? How long did it take? Who was on the team? How many technical resources worked on the implementation? What would you change in retrospect about the implementation?

    This set of questions is useful for accounts with reasonably recent implementations. Don't spend too much time talking about implementation with accounts that are long-standing, as the tool may have changed significantly since then and people forget details anyway. Talk to long-standing accounts about upgrades instead of implementation.

    Getting first-hand accounts of implementations is critical because it's virtually impossible to find out about the implementation process in any other way. (You could instigate a pilot project, but that's a lot of effort and time.) Probe for all details of the implementation, and especially listen for experience and advice on selecting implementation staff and planning for enough time. If the references say to hire an integrator and you're planning for an internal bare-bones project instead, you should rethink your approach.

  • What systems, if any, are integrated with the tool? Why did you decide to integrate them? What effort was required for the integration? Who worked on it? How long did it take? What problems if any are you seeing with the integration today?

    Ideally you want to talk to accounts that have integrations to all the main systems you plan to integrate with. This may require multiple reference checks and they are absolutely worth it, so don't skimp on this.

    Listen carefully for the reasons why the systems are integrated. For one thing, your needs may be different, and some integrations may signal weaknesses in the tool. For instance, are references saying they integrated the tool with a knowledge base system because the one that comes with it is inferior?

  • Who maintains the system today? How many people? What is their background? Is it difficult for you to recruit such individuals? Why? Do you feel that it's a difficult system to maintain? Why? What would you recommend we do when we build our maintenance team?

    The ongoing maintenance of the tool is another area where you need to check references to define your own long-term requirements. Don't bother with these questions with accounts that are very different from yours in terms of scale and technical environment.

    Pay particular attention to the skills required to maintain the system. Will it be difficult for you to attract and retain such individuals?

  • Have you had to upgrade from one version to another? What was the process like? How long did it take? Who performed the upgrade? Did you migrate any customizations? What did you have to do to make it work? Were you forced to upgrade by the vendor? How?

    Upgrades are usually mandated by the vendor on a regular basis, so unless you are planning to use the tool for a very short time, you will simply have to go through the upgrade process. This is a key area when talking with established accounts, as CRM systems have a well-deserved reputation for being difficult to upgrade. Qualify the answers you are getting based on the configuration of the reference account and especially the customizations and integrations they are using.

    Upgrade questions, like implementation questions, are best handled by the IT owners on both sides.

  • Do you have occasion to call the technical support group? How does that work? How quickly do you get a response? How quickly do you get a solution? What support package have you downloaded? When was the last time you had to report a problem? What was it? What was the resolution?

    This is a key question for all reference accounts. Ideally, try to talk to the individuals who are involved in calling the support group, and check that they are working with the same support center as the one you will work with.

    Keep in mind that CRM systems are complex beasts, so problems may take days or weeks to resolve. On the other hand, CRM systems should also run very smoothly so any production failure is a red flag.

  • Have you had occasion to attend training? Was it at the vendor's site or at your site? How long did it take to schedule the training? How would you rate the training? What was the top benefit of the training? The top improvement you would ask for?

    Make sure you make a distinction between end-user training and technical training. Many vendors provide training for technical staff only, leaving the end-user training to the integrator since it's almost always customized.

    Try to talk to the individuals who attended the training to ascertain the quality of it. If they attended training recently, ask them about waiting lists: if you can't train your staff promptly, the implementation schedule will suffer.

  • Do you participate in the user group? In what way? What would you say is valuable about the user group? What would you like to see changed?

    An active user group is a great help in getting information flowing to and from the vendor. Try to figure out who controls the user group: is it pretty much the vendor? Integrators and consultants (this is not rare, nor is it necessarily a bad thing)?

    Is there more to the user group than a yearly meeting with a lot of marketing hoopla? Being able to keep in touch with users year-round is very handy.

  • What do you like best about the tool?

    It's good if references identify items of interest to you in this category.

  • In what areas do you feel the tool is weak? Why? What are the top 3 changes you would like to see made in the product?

    If any items here match key requirements for you, beware!

  • What ROI did you get with the tool?

    Although it's difficult to get candid answers to this question since references consider, rightly, that financial matters are confidential, try to get enough information to evaluate your own ROI goals. If references cite longish periods to get to ROI and yours is optimistic, you should re-evaluate your plans unless your implementation is a lot simpler than theirs (and, if so, maybe they are not such good references for you).

  • Have you ever asked for a new feature? What is the process to do that? Was the feature integrated into the system?

    References often have a much better experience with feature requests than regular accounts since they have a closer relationship with the vendor. Your mileage will vary here, and not for the better. It's a good sign if there is a process to request new features, typically through the technical support group.

  • How do you find out about upcoming features? Do new features match your needs? In what way?

    This is about new features in general, not just the ones requested by the account. The more open the new feature process, the better.

Seeing a Demo

You wouldn't buy a tool without seeing a demo, would you? Yet, many people check references without seeing their implementation of the tool. Demos are much harder to set up than a simple conference call and they take more time but they are invaluable to evaluate customizations and end-user performance. Demos can be performed as a part of a site visit, but a simpler and cheaper alternative is to hold the demo through a web conferencing tool that supports app sharing. Regardless of the medium, references may be reluctant to show what they consider to be confidential information—which they may consider to be both the app and the data, neither of which is a problem in a typical vendor demo. You may be asked to sign a non-disclosure agreement for the demo. In part because of confidentiality concerns, you will most probably be limited to watching your contact work with the app rather than driving it yourself. However, you can and should make requests on what you will see and ask questions as you go. You can refer to the guidelines in the previous chapter about getting good vendor demos: the principles are the same. If you want to see the customer portal piece of the app, you may be able to access it freely on the company's web site. If there is a password-protected area, however, you will need to specifically request access from the reference contact. A final note about checking references. Your project is as unique as your requirements. The fact that a reference was successful or not with a particular area of the tool doesn't mean that the same will be true for you. Do the reference check seriously, but don't let your decision be made solely on the basis of it. You need to have a solid evaluation process independent of the reference check.